Skip to main content

View Post [edit]

Poster: Allen Hsieh Date: Dec 21, 2009 5:01am
Forum: forums Subject: Checking out items via FTP?

Hi I just had a quick question today,
I uploaded a video about two days ago and recently noticed it skipped during certain sections. I am used to the old system of checking out items to edit but when I went through my usual steps I noticed that we are sent to an item editing page (i'm not talking about the meta data/item manager but the new flash item editing page).

Is there any way I can check out the file and use ftp to upload my re-encoded video instead of using the flash uploader? The reason I ask is because my connection is not always stable and the file i'm uploading is about 1.3 gigs. FTP allows me to resume should anything happen.

Thank you so much for your time,
Allen

Reply [edit]

Poster: h4ck3rm1k3 Date: Dec 21, 2009 6:01pm
Forum: forums Subject: Re: Checking out items via FTP?

I second that,

I need to be able to upload via FTP,
my browser crashed after trying to upload 1000 small files via flash.

This is forcing me to create a new item instead of adding to an existing one.

mike

Reply [edit]

Poster: tracey pooh Date: Dec 22, 2009 9:26am
Forum: forums Subject: Re: Checking out items via FTP?

any (full) files it managed to successfully upload pre crash would have
remained on our servers for you to resume adding more to later.

so if you saw none on another attempt, 0 made it up successfully.
i suggest adding maybe 50-100 at a time with each button press/flash submit until they are all there and then submit your new item for creation, etc.

Reply [edit]

Poster: h4ck3rm1k3 Date: Dec 22, 2009 9:37am
Forum: forums Subject: Re: Checking out items via FTP?

I have thousands of files.
It is crazy to use flash or anything like that.
What is wrong with ftp?
mike

Reply [edit]

Poster: tracey pooh Date: Dec 22, 2009 9:46am
Forum: forums Subject: Re: Checking out items via FTP?

we don't work superwell with 1000s of files just so you know.
you'll likely end up with some pretty slow items.

but let's say you want all those files anyway in *1* item.

just use the FTP uploader for the new item.

if you have to change 1000s of files in the item later, the HTTP editing is the old way -- and BTW "flash editor/uploader" ia a wrong misnomer -- people are able to opt to not use flash and instead use normal HTTP POST (ie: a normal form-based method) if they don't want to use the flash widget that monitors file progress.

however, admittedly, doing 1000s of http edits is going to be excruciating.
as a site, we're not designed for such kind of extreme usage. we work with it, but obviously it's not going to be easy for the user/uploader.

a final possible option is S3, but requires more advanced tech ability and command line for the the uploader.

can you post an example item where you have 1000s of files?

--tracey

Reply [edit]

Poster: h4ck3rm1k3 Date: Dec 22, 2009 9:51am
Forum: forums Subject: Re: Checking out items via FTP?

Sure, here is the mail I sent :
Hi there,

I have started to produce openstreetmap files for the archive.

For example converted data from the EPA and census buro.

http://www.archive.org/details/ZCTA_NJ
http://www.archive.org/details/OpenstreetmapZipcodes
http://www.archive.org/details/EPADatapointsOpenStreetmap


My problem is that I have lots of small files, and would like to
update them with FTP.

Also, what I really need is some form of GIT/BZR/HG distributed
version control system for editing these files.
they will be updated on a regular basis.

The data is publicdomain/creative commons, but archive.org is not
ideal for processing them.

Are you interested in working with me to making a better way to host
map data on archive org?
I am a programmer and would like to work with you on this.
thanks,
mike

Reply [edit]

Poster: tracey pooh Date: Feb 18, 2010 2:02pm
Forum: forums Subject: Re: Checking out items via FTP?

and once again, lots of small files = fail with our site.

we have some interesting tools/solutions for that,
including an automatic and efficient 'zipview' utility
that we could try to setup for your items.
this allows a single file to be pulled out a of zip to an end user.
--t

Reply [edit]

Poster: h4ck3rm1k3 Date: Dec 22, 2009 9:56am
Forum: forums Subject: Re: Checking out items via FTP?

The other thing that I would like to say is that for these zip code files we need to have small individual files, because people will only access one or two at a given time.
To put them all into a zip file would defeat the purpose, unless there is a way to access only a certain offset in the file some how.

Reply [edit]

Poster: tracey pooh Date: Feb 18, 2010 2:01pm
Forum: forums Subject: Re: Checking out items via FTP?

the only robo option for something like that that we have available is S3.
that would allow for all of what you want i think..
-t

Reply [edit]

Poster: tracey pooh Date: Dec 22, 2009 9:29am
Forum: forums Subject: Re: Checking out items via FTP?

we don't support FTP reuploading, adding, or changing of files
in *existing* items.

and our FTP support for *new* items does *not* support resumption, anyway, unfortunately, so the main advantage (resumption) you'd want FTP for wouldn't work with our site, anyway 8-(

--tracey, archive staff

Reply [edit]

Poster: Allen Hsieh Date: Dec 22, 2009 11:39am
Forum: forums Subject: Re: Checking out items via FTP?

Thank you for the heads up Tracey!

Another reason I was asking though was because when i use the flash uploader, try the "prefer not to use flash?" option, or use the alternative (non port 8080) page to upload it appears that my file never gets completed.

In the case of the flash uploader, it will say the file has transferred and completed within 4 minutes (the file is 1.3 gigs and I only dream of my connection ever being that fast, haha [usual FTP upload speed is around 200kb/s])

For the other two methods, it just hangs there. I've tried leaving the computer alone for 3 hours for each method to see what will happen and still no luck.

Is there anyway I could delete the item so I can re-use the identifier and try uploading it anew? The identifier in question is 12.19.09_TitleFight > http://www.archive.org/details/12.19.09_TitleFight

Sorry for all the headaches and again, thank you so much for your time in providing support for a wonderful free service,
Allen

Reply [edit]

Poster: tracey pooh Date: Feb 1, 2010 5:44pm
Forum: forums Subject: Re: Checking out items via FTP?

ok i've cleared out that item so that you can try again.

for a 1.3G file, depending on where you are, i'd be surprised if it doesn't take more like 5+ hours or more. maybe even like 24+? :(

i would try doing some smaller files first to the item (you can delete the files later) and make sure that is working. maybe start with 10MB, then 50MB, then 150MB and if they all work, gauge how long 1.3G might take given the 15OMB file.

other thing worth noting is that carriers like comcast *definitely* will "shape" your upload speed. that is, once X seconds go by, they will automatically drop your upload rate dramatically. it really peeves me off but it's true...
--tracey